Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experimental: Use shared memory pool even when offHeap is not enabled #998

Closed
wants to merge 12 commits into from

Conversation

andygrove
Copy link
Member

Which issue does this PR close?

Closes #996

Rationale for this change

We currently create separate memory pools for each native plan within an executor and we do not take concurrency into account when executors have multiple cores assigned, making this difficult to configure.

What changes are included in this PR?

Always use a shared memory pool per executor.

How are these changes tested?

@andygrove andygrove closed this Oct 7, 2024
@andygrove andygrove reopened this Oct 7, 2024
@andygrove andygrove closed this Oct 7, 2024
@andygrove andygrove deleted the shared-mempool branch October 7, 2024 20:19
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 50.00000% with 13 lines in your changes missing coverage. Please review.

Project coverage is 34.20%. Comparing base (c3023c5) to head (34596c5).
Report is 24 commits behind head on main.

Files with missing lines Patch % Lines
.../java/org/apache/spark/CometTaskMemoryManager.java 40.90% 10 Missing and 3 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #998      +/-   ##
============================================
+ Coverage     34.03%   34.20%   +0.16%     
- Complexity      875      883       +8     
============================================
  Files           112      112              
  Lines         43289    43442     +153     
  Branches       9572     9625      +53     
============================================
+ Hits          14734    14859     +125     
+ Misses        25521    25519       -2     
- Partials       3034     3064      +30     
Flag Coverage Δ
?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement shared memory pool when native memory management is enabled
2 participants